Digital cameras and other image capture devices employ image sensors, such as charge coupled devices (CCDs), complementary metal oxide semiconductor (CMOS) devices or Foveon sensors, to capture images for subsequent storage and display. A typical CCD or CMOS color image sensor in an image capture device comprises a grid or array of photosensitive locations commonly referred to as photosites. During image capture of a scene, each photosite is capable of sensing only a single green, red or blue color. The resulting digital image data forms a pattern (or “mosaic”) of red, green and blue colors known as a color filter array (CFA). In order to reproduce the scene on a visual display in proper color, each pixel in the visual display requires full color information. As will be appreciated, the CFA is therefore missing information required to reproduce the scene in proper color.
To produce a full color image from a CFA, green, red and blue colors must be interpolated at photosites at which they are missing (i.e., have not been sensed), using sensed colors at local photosites. The process of interpolating missing colors in a CFA is known as de-mosaicing.
The Bayer pattern, a well-known CFA, is a two-by-two array of colors half of which are green, a quarter of which are red, and a quarter of which are blue. Rows of the Bayer CFA have alternating green and red photosites, or alternating green and blue photosites, whereby the green photosites in the Bayer CFA are distributed in a quincunx pattern. Methods for de-mosaicing the Bayer CFA and other CFAs are very well documented. For example, U.S. Pat. No. 6,421,084 to Chang et al. discloses a method for recovering the missing colors in a captured Bayer color filter array. During the method, a set of gradient values is determined for each data element at a specific location in the color filter array. The gradient values correspond to color value differences in a plurality of directions from the data element. From the set of gradient values, a threshold value is determined. Using the threshold value, a subset of the gradient values is selected so that the members of the subset have gradient values less than the threshold value. Additional color values for each data element are interpolated on the basis of the subset of gradient values, rather than in only a single direction.
U.S. Pat. No. 6,236,430 to Suzuki et al. discloses an image capture device capable of moving a Bayer filter in several directions relative to a charge coupled device (CCD) in order to capture multiple sets of image data for each image. The multiple sets of data comprising red, green and blue information for each sensor position in the color filter array are synthesized in order to output a full color image without interpolation.
U.S. Pat. No. 6,496,608 to Chui discloses an image data interpolation system and method that employs finite impulse response filters. Based on a subset of captured Bayer pattern CFA values and a pixel filling filter equation, smoothness, low distortion and continuous surfaces are ensured.
U.S. Pat. No. 6,836,289 to Koshiba et al. discloses a method for interpolating a Bayer color filter array. First the green channel is interpolated to yield a green plane. After the green plane has been obtained, the edges in the green plane are detected and used to compute a high-pass component of the green channel. Red and blue interpolation is performed and the results are corrected using the weighted high-pass component of the green channel.
U.S. Pat. No. 6,900,836 to Hamilton, Jr. discloses a method for correcting corruption in a column of pixels in an image sensor. Classifiers are computed using non-corrupt pixel data adjacent to corrupt pixel data in order to determine whether either horizontal or diagonal edges run through the column. Interpolation is conducted first for green and then for red/blue corrupted pixels in the column according to the edge direction indicated by the classifiers.
U.S. Patent Application Publication Nos. 2002/0063789 and 2005/0174441 to Acharya et al. disclose a color filter array that simplifies the process of interpolating unsensed color values. The CFA comprises more than half green sensors, thereby enabling an interpolation scheme to more accurately result in a full green channel. Edge zones, smooth zones and stripe zones are computed by determining the variance in values in 3×3 arrays of pixel values. Interpolation is conducted based on the type of zone computed in each array.
U.S. Patent Application Publication No. 2002/0167602 to Nguyen discloses a method for de-mosaicing image data captured according to the Bayer CFA that purports to reduce interpolation artifacts along feature edges. Color discontinuities are equalized on the assumption that the changes in local color intensity values relative to a local average are the same for each of the color components.
U.S. Patent Application Publication No. 2003/0052981 to Kakarala et al. discloses a method for interpolating missing color values in raw data captured as a color filter array. The method employs a voting scheme for determining an interpolation direction or combination of directions for each of the missing pixel locations. The interpolation is conducted using color differencing for first the luminance color plane, and then the chrominance color planes based on the interpolated and sensed luminance data. The luminance and chrominance color planes are then summed to yield an output image.
U.S. Patent Application Publication No. 2003/0117507 to Kehtarnavaz et al. discloses a color filter array interpolation scheme that uses directional derivatives to determine interpolation directions for each missing pixel. The pixel values surrounding a missing pixel are weighted inversely during interpolation with the directional derivatives in vertical, horizontal and diagonal directions in order to yield an interpolated value. First, luminance interpolation is conducted. Next, chrominance interpolation is conducted based on the luminance interpolation. Chrominance interpolation is conducted by interpolating red at sensed blue and blue at sensed red using the weights previously determined, and then red at sensed green and blue at sensed green using direction predictors.
U.S. Patent Application Publication No. 2004/0051798 to Kakarala et al. discloses a method for correcting corrupt pixels in a digital color image sensor. For a current pixel, gradients of neighboring pixels in at least one other color plane than the color plane of the current pixel are used to determine whether the current pixel is defective. Both the gradients and a range of sensor values from neighboring pixels in the same color plane as the current pixel can be determined. If the sensor value of the current pixel is outside of the range by a threshold amount that is calculated using one or more of the gradients, the current pixel is determined to be a defective pixel. The gradients in other color planes are said to provide a more accurate prediction of the sensor value of the current pixel by considering the local roughness in the image.
U.S. Patent Application Publication No. 2004/0141072 to Nilsson et al. discloses a method of interpolating colors in raw data captured using a Bayer CFA. Gradient information is collected for the four compass directions from both the luminance channel and chrominance channel that is to be interpolated, and an inverse gradient function is applied in order to provide greater weight to interpolation along edges. Interpolation is performed using color differencing.
U.S. Patent Application Publication No. 2004/0161145 to Embler discloses a method for interpolating color values captured using a Bayer CFA based on color samples of nearby locations. The interpolated values are selectively adjusted based on correlations between pre-identified reference patterns of values and the patterns of values within neighborhoods of the pixels. The reference patterns may be combinations of a high reference value and a low reference value such as maximum and minimum color values of the digitized video signal. The set of reference patterns represents various luminance gradients within the range.
U.S. Patent Application Publication No. 2004/0169747 to Ono et al. discloses a method for directionally interpolating color channel values in a Bayer CFA. Weights are computed based on averages of pixel values adjacent a location of interest, and are used to directionally interpolate the pixel value at the location of interest.
U.S. Patent Application Publication No. 2004/0179752 to Cheng et al. discloses a system and method for interpolating a color image from an array of single color sensors. For each data element in the array, four gradients are computed that specify the color and/or luminance difference along different linear paths passing through the data element. The gradients are calculated using a first-order color differential and a second order (Laplacian) color or luminance differential along the respective linear path. Interpolation is also conducted using first and second order color differentials in a five-by-five array of pixels surrounding the current pixel.
U.S. Patent Application Publication No. 2005/0058361 to Tajima discloses an interpolation method for a Bayer pattern CFA. Unsensed colors are interpolated based on a weighted average of the outputs of vertical and horizontal interpolation units. The weights are determined by computation of high frequency components in the image and are varied in order to effectively vary the direction of interpolation as required from vertical to horizontal and various degrees of slope in-between.
U.S. Patent Application Publication No. 2005/0146629 to Muresan discloses an edge directed de-mosaicing method for a CFA that sequentially employs a coarse model fit for determining edge direction, a color-difference interpolation scheme in the edge direction, and then a fine model fit, which uses the interpolated values to re-estimate the local edge direction.
U.S. Patent Application Publication No. 2005/0169521 to Hel-Or discloses a method for reducing aliasing and zippered edges caused during the process of de-mosaicing raw data captured using a Bayer CFA. A bilinear or bicubic interpolation is used to estimate missing information in each of the green, red and blue color planes, and the full color image is transformed to luminance-chrominance-chrominance color space. Smoothing is performed on the transformed image, and then the image is transformed back to the red-green-blue (RGB) color space.
U.S. Pat. No. 5,382,976 to Hibbard discloses a method for adaptively interpolating a full color image that uses luminance gradients to determine an interpolation direction. Gradient values for the luminance channel are determined in at least two directions and compared to a threshold. One of the directions is chosen based on the gradients and the threshold, as the direction in which first unsensed greens are interpolated. Red and blue values are then interpolated without consideration of direction, using color differencing.
U.S. Pat. No. 5,627,734 to Hamilton Jr. et al. discloses a sequential adaptive directional interpolation scheme for colors in a Bayer CFA. A first interpolation direction is selected using a classifier computed by adding LaPlacian second-order values of the green plane to gradient values between known chrominance pixel locations surrounding an unknown green pixel. Then, the selected interpolation direction is used to perform the green interpolation and a correction factor is added based on either neighboring red or blue pixels. After green interpolation, chrominance interpolation that employs the selected interpolation direction is used to interpolate chrominance values and a correction factor is then added to the interpolated chrominance value.
U.S. Pat. No. 6,570,616 to Chen discloses a method for de-mosaicing image data captured using a Bayer pattern CFA that detects edges in the data by, for green interpolation, comparing a primary reference pixel adjacent to a target location of interest with secondary reference pixels surrounding the primary reference pixel. Where the primary reference pixel has a greater luminance than all surrounding pixels, an edge is detected. For example, an edge in the vertical direction is detected for the target location when a value associated with the primary reference pixel located at the top side of the target location is larger than the corresponding values associated with the four secondary reference pixels existing in the vicinity of the primary reference pixel and when a value associated with the primary reference pixel at the bottom side of the target location is larger than the corresponding values associated with the four secondary reference pixels existing in the vicinity of the primary reference pixel. When relative evaluation of primary and secondary reference pixels is reversed, an edge is also considered to have been detected. Green interpolation is performed in accordance with the determined edge direction using only green pixels, and red and blue channels are then interpolated without explicitly employing edge information, using color differencing.
U.S. Pat. No. 6,714,232 to Kobayashi discloses a method for interpolating missing color data captured using a Bayer arrangement of red, green and blue colors. A correlation measurement is obtained in eight directions relative to a target pixel by determining differences between pixels in each direction. The correlation measurements are used as coefficients for weighting the contribution of each direction to the interpolation. Interpolation is then performed in one of the vertical, horizontal and diagonal directions.
U.S. Pat. No. 6,744,916 to Takahashi discloses an image processing apparatus and method for interpolating missing pixels in a color filter array. The horizontal and vertical gradients for each of four adjacent pixels around a missing pixel are calculated. Based on the gradients, the minimum for each horizontal and vertical set is determined and used in combination with horizontal and vertical thresholds to yield the missing pixel value. Diagonal interpolation is conducted using combinations of the vertical and horizontal values. The combinations are based on the horizontal and vertical gradients for the missing pixel.
U.S. Pat. No. 6,833,868 to Meynants et al. discloses an interpolation method for color filter arrays that compares values of chrominance in the vicinity of a missing pixel and using a vote-operator, employs the dominant median or average chrominance value to be used for interpolation. Interpolation is conducted by averaging neighboring pixels.
U.S. Pat. No. 6,897,425 to Osada discloses an interpolation method for color filter arrays which employs a simple absolute difference of horizontal and vertical intensities to determine whether there is any correlation therebetween. Correlation is established if a threshold is exceeded. Interpolation is performed after conversion to the YCbCr color space based on the correlation measurements.
U.S. Pat. Nos. 6,900,833 and 6,747,698 to Abe disclose an image interpolating device that uses the absolute difference of horizontal and vertical intensities to determine an interpolation direction for missing pixels in a color filter array. The green channel in the CFA is interpolated according to the determined direction information, and the red/blue interpolation at sensed blue/red locations is then conducted without explicit use of direction information. Chromatic blur is reduced by conducting red/blue interpolation at sensed green locations after conversion to the YCbCr colorspace.
U.S. Patent Application Publication No. 2004/0095475 to Yamamoto et al. discloses a method for interpolating missing pixel values in a color filter array that calculates both horizontal and vertical estimates for missing RGB values. A correlation template is then employed to compute normalized weights to vary the contribution of each of the estimates to the interpolated value. Horizontal and vertical green values are calculated using a first weight template to interpolate green on unknown green and to refine green on known green. Vertical and horizontal red and blue values are calculated using a color difference method.
U.S. Patent Application Publication No. 2004/0160521 to Yamamoto discloses an image processing device that interpolates missing values in a color filter array. The device calculates directional strength using gradients and allots weights obtained by experiment in the form of coefficients for the interpolation in multiple directions. In accordance with whether the determined directional strength lies in the negative or positive range, directional interpolation of green pixels is applied using neighborhood sensed green and red/blue pixels. This method is similar to that disclosed in above-mentioned U.S. Pat. No. 5,629,734.
U.S. Patent Application Publication No. 2005/0007470 to Kimura et al. discloses an interpolation method for color filter arrays that employs a template-based gradient calculation method. High frequency edge directions are determined using the templates, and interpolation is performed in accordance with the edge detection using proximate same-channel values.
While methods of interpolating missing color values in CFAs are well documented, the prior art references discussed above disclose techniques that tend to suffer from degradation in image quality. Degradation is caused by inaccuracies due to the selection of a sub-optimal interpolation direction or the introduction of artifacts during interpolation. Furthermore, many of the prior art techniques discussed above require complex and costly processing for second or higher order computation. Improvements are therefore desired.
It is therefore an object to provide a novel method and apparatus for interpolating missing colors in a color filter array.